<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
    <head>
        <title>RVM : Experimental Guidelines</title>
	    <link rel="stylesheet" href="styles/site.css" type="text/css" />
        <META http-equiv="Content-Type" content="text/html; charset=UTF-8">	    
    </head>

    <body>
	    <table class="pagecontent" border="0" cellpadding="0" cellspacing="0" width="100%" bgcolor="#ffffff">
		    <tr>
			    <td valign="top" class="pagebody">
				    <div class="pageheader">
					    <span class="pagetitle">
                            RVM : Experimental Guidelines
                                                    </span>
				    </div>
				    <div class="pagesubheading">
					    This page last changed on May 29, 2009 by <font color="#0050B2">pizlo</font>.
				    </div>

				    <p>This section provides some tips on collecting performance numbers with Jikes RVM.</p>

<h2><a name="ExperimentalGuidelines-WhichbootimageshouldIuse%3F"></a>Which boot image should I use?</h2>

<p>To make a long story short the best performing configuration of Jikes RVM will almost always be <tt>production</tt>.  Unless you really know what you are doing, don't use any other configuration to do a performance evaluation of Jikes RVM.</p>

<p>Any boot image you use for performance evaluation must have the following characteristics for the results to be meaningful:</p>
<ul>
	<li>config.assertions=none. Unless this is set, the runtime system and optimizing compiler will perform fairly extensive assertion checking. This introduces significant runtime overhead. By convention, a configuration with the Fast prefix disables assertion checking.</li>
	<li>config.bootimage.compiler=opt. Unless this is set, the boot image will be compiled with the baseline compiler and virtual machine performance will be abysmal. Jikes RVM has been designed under the assumption that aggressive inlining and optimization will be applied to the VM source code.</li>
</ul>


<h2><a name="ExperimentalGuidelines-WhatcommandlineargumentsshouldIuse%3F"></a>What command-line arguments should I use?</h2>

<p>For best performance we recommend the following:</p>
<ul>
	<li><tt>&#45;X:processors=all</tt>: By default, Jikes™ RVM uses only one processor for garbage collection. Setting this option tells the garbage collection system to utilize all available processors.</li>
	<li>Set the heap size generously.  We typically set the heap size to at least half the physical memory on a machine.</li>
</ul>


<h2><a name="ExperimentalGuidelines-%26nbsp%3BCompilerReplay"></a>&nbsp;Compiler Replay</h2>

<p>The compiler-replay methodology is deterministic and eliminates memory allocation and mutator variations due to non-deterministic application of the adaptive compiler. We need this latter methodology because the non-determinism of the adaptive compilation system makes it a difficult platform for detailed performance studies. For example, we cannot determine if a variation is due to the system change being studied or just a different application of the adaptive compiler. The information we record and use are hot methods and blocks information. We also record dynamic call graph with calling frequency on each edge for inlining decisions.<br/>
Here is how to use it:</p>
<ol>
	<li>Generate the profile information, using the following command line arguments:
<h4><a name="ExperimentalGuidelines-Foredgeprofile"></a>For edge profile</h4>
<div class="panel" style="border-width: 1px;"><div class="panelContent">
<p>&#45;X:base:edge_counters=true<br/>
&#45;X:base:edge_counter_file=my_edge_counter_file</p>
</div></div>
<h4><a name="ExperimentalGuidelines-Foradaptivecompilationprofile"></a>For adaptive compilation profile</h4>
<div class="panel" style="border-width: 1px;"><div class="panelContent">
<p>&#45;X:aos:enable_advice_generation=true<br/>
&#45;X:aos:cafo=my_compiler_advice_file</p>
</div></div>
<h4><a name="ExperimentalGuidelines-Fordynamiccallgraphprofile%28usedbyadaptiveinlining%29"></a>For dynamic call graph profile (used by adaptive inlining)</h4>
<div class="panel" style="border-width: 1px;"><div class="panelContent">
<p>&#45;X:aos:dcfo=my_dynamic_call_graph_file<br/>
&#45;X:aos:final_report_level=2</p>
</div></div>
<p>Typically you might run a benchmark several times and choose the set of replay data that produced the best performance.
<br clear="all" /></p></li>
	<li>Use the profile you generated for compiler replay, using the following command line arguments:<br/>
<div class="panel" style="border-width: 1px;"><div class="panelContent">
<p>&#45;X:aos:enable_replay_compile=true<br/>
&#45;X:vm:edgeCounterFile=my_edge_counter_file<br/>
&#45;X:aos:cafi=my_compiler_advice_file<br/>
&#45;X:aos:dcfi=my_dynamic_call_graph_file</p>
</div></div><br clear="all" /></li>
</ol>


<h2><a name="ExperimentalGuidelines-MeasuringGCperformance"></a>Measuring GC performance</h2>

<p>&nbsp;MMTk includes a statistics subsystem and a harness mechanism for measuring its performance.&nbsp; If you are using the DaCapo benchmarks, the MMTk harness can be invoked using the '-c MMTkCallback' command line option, but for other benchmarks you will need to invoke the harness by calling the static methods</p>
<div class="panel" style="border-width: 1px;"><div class="panelContent">
<p>org.mmtk.plan.Plan.harnessBegin()<br/>
org.mmtk.plan.Plan.harnessEnd()</p>
</div></div>
<p>at the appropriate places.&nbsp; Other command line switches that affect the collection of statistics are</p>
<table class='confluenceTable'><tbody>
<tr>
<th class='confluenceTh'> Option </th>
<th class='confluenceTh'> Description </th>
</tr>
<tr>
<td class='confluenceTd'> &#45;X:gc:printPhaseStats=true </td>
<td class='confluenceTd'> Print statistics for each mutator/gc phase during the run </td>
</tr>
<tr>
<td class='confluenceTd'> &#45;X:gc:xmlStats=true </td>
<td class='confluenceTd'> Print statistics in an XML format (as opposed to human-readable format) <br clear="all" /> </td>
</tr>
<tr>
<td class='confluenceTd'> &#45;X:gc:verbose <br clear="all" /> </td>
<td class='confluenceTd'> This is incompatible with MMTk's statistics system. <br clear="all" /> </td>
</tr>
<tr>
<td class='confluenceTd'> &#45;X:gc:variableSizeHeap=false </td>
<td class='confluenceTd'> Disable dynamic resizing of the heap <br clear="all" /> </td>
</tr>
</tbody></table>
<p>Unless you are specifically researching flexible heap sizes, it is best to run benchmarks in a fixed size heap, using a range of heap sizes to produce a curve that reflects the space-time tradeoff.&nbsp; Using replay compilation and measuring the second iteration of a benchmark is a good way to produce results with low noise.</p>

<p>There is an active debate among memory management and VM researchers about how best to measure performance, and this section is not meant to dictate or advocate any particular position, simply to describe one particular methodology.</p>

<h2><a name="ExperimentalGuidelines-JikesRVMisreallyslow%5C%21WhatamIdoingwrong%3F"></a>Jikes RVM is really slow&#33; What am I doing wrong?</h2>

<p>Perhaps you are not seeing stellar Jikes™ RVM performance. If Jikes RVM as described above is not competitive product JVMs, we recommend you test your installation with the DaCapo benchmarks.  We expect Jikes RVM performance to be very close to Sun's HotSpot 1.5 server running the DaCapo benchmarks (see our <a href="http://jikesrvm.anu.edu.au/~dacapo/release/index.php">Nighlty DaCapo performance comparisions</a>page for the daily data). &nbsp; Of course, running DaCapo well does not guarantee that Jikes RVM runs all codes well.</p>

<p>Some kinds of code will not run fast on Jikes RVM.  Known issues include:</p>
<ol>
	<li>Jikes RVM start-up may be slow compared to the some product JVMs.</li>
	<li>Remember that the non-adaptive configurations (-X:aos:enable_recompilation=false &#45;X:aos:initial_compiler=opt) opt-compile <em>every</em> method the first time it executes.  With aggressive optimization levels, opt-compiling will severely slow down the first execution of each method.  For many benchmarks, it is possible to test the quality of generated code by either running for several iterations and ignoring the first, or by building a warm-up period into the code.  The SPEC benchmarks already use these strategies.  The adaptive configuration does not have this problem; however, we cannot stipulate that the adaptive system will compete with the product on short-running codes of a few seconds.</li>
	<li>Performance on tight loops may suffer.  The Jikes RVM mechanism for safe points (thread preemption for garbage collection, on-stack-replacement, profiling, etc) relies on the insertion of a yield test on every back edge.  This will hurt tight loops, including many simple microbenchmarks.  We should someday alleviate this problem by strip-mining and hoisting the yield point out of hot loops, or implementing a safe point mechanism that does not require an explicit check.</li>
	<li>The load balancing in the system is naive and unfair. This can hurt some styles of codes, including bulk-synchronous parallel programs.</li>
</ol>


<p>The Jikes RVM developers wish to ensure that Jikes RVM delivers competitive performance.  If you can isolate reproducible performance problems, please let us know. </p>

				    
                    			    </td>
		    </tr>
	    </table>
	    <table border="0" cellpadding="0" cellspacing="0" width="100%">
			<tr>
				<td height="12" background="http://docs.codehaus.org/images/border/border_bottom.gif"><img src="images/border/spacer.gif" width="1" height="1" border="0"/></td>
			</tr>
		    <tr>
			    <td align="center"><font color="grey">Document generated by Confluence on Jun 10, 2009 09:12</font></td>
		    </tr>
	    </table>
    </body>
</html>